{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Using `pymldb`'s Progress Bar and Cancel Button Tutorial\n",
    "\n",
    "This tutorial showcases the use of progress bars and cancel buttons for long-running procedures with `pymldb` with a Jupyter notebook. This allows a user to see the progress of a procedure as well as cancel it.\n",
    "\n",
    "If you have not done so already, we encourage you to go through the [Using `pymldb` Tutorial](../../../../doc/nblink.html#_tutorials/Using pymldb Tutorial).\n",
    "\n",
    "## How does it work?\n",
    "\n",
    "To use this feature, you only need to slightly modify the way you execute procedures. For example, when doing an HTTP PUT, you would go from using `mldb.put()` to `mldb.put_and_track()`.\n",
    "\n",
    "The cancel button is displayed as soon as the procedure run id is found. The button is removed as soon as the procedure finishes either normally or with an error.\n",
    "\n",
    "The progress bar library used is [tqdm/tqdm](https://github.com/tqdm/tqdm). Progress bars are displayed as soon as a procedure enters the \"executing\" state. Then they are refreshed at every interval for as long as the procedure stays in the \"executing\" state. They move to a valid state (they turn green) when a step/procedure finishes normally and to a danger state (they turn red) when they finish with an error.\n",
    "\n",
    "If a procedure runs too quickly, the progress bars will not be displayed because the application logic will not have time to catch the \"executing\" phase. If a procedure stays in the \"initializing\" phase for some time, the \"Cancel\" button will be visible with no progress bars as long as the \"executing\" phase is not reached.\n",
    "\n",
    "## ⚠ Disclaimers\n",
    "1. There is a known issue where the final value of the last progress bar may not reflect the real final value of what was done in MLDB. The reason for it is that once a procedure has finished running, it no longer reports how many items it processed for each step.\n",
    "2. Due to XSS (cross site scripting) restrictions, the cancel button provided with the progress bars will not work if the notebook is running on a different host than mldb itself.\n",
    "\n",
    "Here we start with the obligatory lines to import pymldb and initialize the connection to MLDB."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pymldb\n",
    "mldb = pymldb.Connection()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Procedure with steps\n",
    "Here we post to a procedure with multiple steps. The steps are displayed as soon as the procedure starts running and are updated accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "                            <script type=\"text/javascript\">\n",
       "                                function cancel_2017_01_24T16_25_13_172311Z_463496b56263af05(btn) {\n",
       "                                    $(btn).attr(\"disabled\", true).html(\"Cancelling...\");\n",
       "                                    $.ajax({\n",
       "                                        url: \"http://localhost:17055/v1/procedures/auto-80cb9c6afbb7ad8f-9d7975535e06e5ff/runs/2017-01-24T16:25:13.172311Z-463496b56263af05/state\",\n",
       "                                        type: 'PUT',\n",
       "                                        data: JSON.stringify({\"state\" : \"cancelled\"}),\n",
       "                                        success: () => { $(btn).html(\"Cancelled\"); },\n",
       "                                        error: (xhr) => { console.error(xhr);\n",
       "                                                            console.warn(\"If this is a Cross-Origin Request, this is a normal error. Otherwise you may report it.\");\n",
       "                                                            $(btn).html(\"Cancellation failed - See JavaScript console\");\n",
       "                                                        }\n",
       "                                    });\n",
       "                                }\n",
       "                            </script>\n",
       "                            <button id=\"2017_01_24T16_25_13_172311Z_463496b56263af05\" onclick=\"cancel_2017_01_24T16_25_13_172311Z_463496b56263af05(this);\">Cancel</button>\n",
       "                        "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                        <script type=\"text/javascript\" class=\"removeMe\">\n",
       "                            $(function() {\n",
       "                                var outputArea = $(\".removeMe\").parents(\".output_area:first\");\n",
       "                                outputArea.prevAll().remove();\n",
       "                                outputArea.next().remove();\n",
       "                                outputArea.remove();\n",
       "                            })\n",
       "                        </script>\n",
       "                    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "<Response [201]>\n"
     ]
    }
   ],
   "source": [
    "print mldb.post_and_track('/v1/procedures', {\n",
    "        'type' : 'mock',\n",
    "        'params' : {'durationMs' : 8000, \"refreshRateMs\" : 500}\n",
    "    }, 0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Procedure with no steps\n",
    "A procedure with no inner steps will simply display its progress.\n",
    "\n",
    "This one is an example where the \"initializing\" phase sticks for some time, so the \"Cancel\" button is shown alone and eventually, when the \"executing\" phase is reached, the progress bar is displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "                            <script type=\"text/javascript\">\n",
       "                                function cancel_2017_01_24T16_26_15_213398Z_463496b56263af05(btn) {\n",
       "                                    $(btn).attr(\"disabled\", true).html(\"Cancelling...\");\n",
       "                                    $.ajax({\n",
       "                                        url: \"http://localhost:17055/v1/procedures/embedded_imagess/runs/2017-01-24T16:26:15.213398Z-463496b56263af05/state\",\n",
       "                                        type: 'PUT',\n",
       "                                        data: JSON.stringify({\"state\" : \"cancelled\"}),\n",
       "                                        success: () => { $(btn).html(\"Cancelled\"); },\n",
       "                                        error: (xhr) => { console.error(xhr);\n",
       "                                                            console.warn(\"If this is a Cross-Origin Request, this is a normal error. Otherwise you may report it.\");\n",
       "                                                            $(btn).html(\"Cancellation failed - See JavaScript console\");\n",
       "                                                        }\n",
       "                                    });\n",
       "                                }\n",
       "                            </script>\n",
       "                            <button id=\"2017_01_24T16_26_15_213398Z_463496b56263af05\" onclick=\"cancel_2017_01_24T16_26_15_213398Z_463496b56263af05(this);\">Cancel</button>\n",
       "                        "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                        <script type=\"text/javascript\" class=\"removeMe\">\n",
       "                            $(function() {\n",
       "                                var outputArea = $(\".removeMe\").parents(\".output_area:first\");\n",
       "                                outputArea.prevAll().remove();\n",
       "                                outputArea.next().remove();\n",
       "                                outputArea.remove();\n",
       "                            })\n",
       "                        </script>\n",
       "                    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "<Response [201]>\n"
     ]
    }
   ],
   "source": [
    "print mldb.put_and_track('/v1/procedures/embedded_imagess', {\n",
    "    'type' : 'import.text',\n",
    "    'params' : {\n",
    "        'dataFileUrl' : 'https://s3.amazonaws.com/benchm-ml--main/train-1m.csv',\n",
    "        'outputDataset' : {\n",
    "            'id' : 'embedded_images_realestate',\n",
    "            'type' : 'sparse.mutable'\n",
    "        }\n",
    "    }\n",
    "}, 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serial procedure\n",
    "When using post_and_track along with a serial procedure, a progress bar is displayed for each step. They will only take the value of 0/1 and 1/1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "                            <script type=\"text/javascript\">\n",
       "                                function cancel_2017_01_24T16_27_28_127061Z_463496b56263af05(btn) {\n",
       "                                    $(btn).attr(\"disabled\", true).html(\"Cancelling...\");\n",
       "                                    $.ajax({\n",
       "                                        url: \"http://localhost:17055/v1/procedures/auto-6e107b854894ea54-6800e009079893fc/runs/2017-01-24T16:27:28.127061Z-463496b56263af05/state\",\n",
       "                                        type: 'PUT',\n",
       "                                        data: JSON.stringify({\"state\" : \"cancelled\"}),\n",
       "                                        success: () => { $(btn).html(\"Cancelled\"); },\n",
       "                                        error: (xhr) => { console.error(xhr);\n",
       "                                                            console.warn(\"If this is a Cross-Origin Request, this is a normal error. Otherwise you may report it.\");\n",
       "                                                            $(btn).html(\"Cancellation failed - See JavaScript console\");\n",
       "                                                        }\n",
       "                                    });\n",
       "                                }\n",
       "                            </script>\n",
       "                            <button id=\"2017_01_24T16_27_28_127061Z_463496b56263af05\" onclick=\"cancel_2017_01_24T16_27_28_127061Z_463496b56263af05(this);\">Cancel</button>\n",
       "                        "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                        <script type=\"text/javascript\" class=\"removeMe\">\n",
       "                            $(function() {\n",
       "                                var outputArea = $(\".removeMe\").parents(\".output_area:first\");\n",
       "                                outputArea.prevAll().remove();\n",
       "                                outputArea.next().remove();\n",
       "                                outputArea.remove();\n",
       "                            })\n",
       "                        </script>\n",
       "                    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "<Response [201]>\n"
     ]
    }
   ],
   "source": [
    "prefix = 'http://public.mldb.ai/datasets/dataset-builder'\n",
    "print mldb.post_and_track('/v1/procedures', {\n",
    "    'type' : 'serial',\n",
    "    'params' : {\n",
    "        'steps' : [\n",
    "            {\n",
    "                'type' : 'mock',\n",
    "                'params' : {'durationMs' : 2000, \"refreshRateMs\" : 500}\n",
    "            }, {\n",
    "                'type' : 'import.text',\n",
    "                'params' : {\n",
    "                    'dataFileUrl' : prefix + '/cache/dataset_creator_embedding_realestate.csv.gz',\n",
    "                    'outputDataset' : {\n",
    "                        'id' : 'embedded_images_realestate',\n",
    "                        'type' : 'embedding'\n",
    "                    },\n",
    "                    'select' : '* EXCLUDING(rowName)',\n",
    "                    'named' : 'rowName',\n",
    "                }\n",
    "            }, {\n",
    "                'type' : 'mock',\n",
    "                'params' : {'durationMs' : 2000, \"refreshRateMs\" : 500}\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Where to next?\n",
    "\n",
    "Check out the other [Tutorials and Demos](../../../../doc/#builtin/Demos.md.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  },
  "widgets": {
   "state": {
    "1547f39f55a3486992b25a60e84b2f31": {
     "views": [
      {
       "cell_index": 3
      }
     ]
    },
    "378419543b0942d0aa39f8e211e58d2f": {
     "views": [
      {
       "cell_index": 7
      }
     ]
    },
    "4c835ec1cf7c46db9aba344f3522fa35": {
     "views": [
      {
       "cell_index": 7
      }
     ]
    },
    "7edf3ab4d99c4220a7b0a7aa2f0a4fe8": {
     "views": [
      {
       "cell_index": 7
      }
     ]
    },
    "8abf2a674ad04e37808690633c14612f": {
     "views": [
      {
       "cell_index": 7
      }
     ]
    },
    "c879fb1e34b14526bc7c89aadaf3a76e": {
     "views": [
      {
       "cell_index": 5
      }
     ]
    },
    "dc1f79109d5f44599427b1e47a3745fe": {
     "views": [
      {
       "cell_index": 3
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
